Repository navigation
perf: Improve performance of md5 - #19568
Merged
Merged
Conversation
comphead
reviewed
Dec 30, 2025
| /// Fast hex encoding using a lookup table instead of format strings. | ||
| /// This is significantly faster than using `write!("{:02x}")` for each byte. | ||
| #[inline] | ||
| fn hex_encode<T: AsRef<[u8]>>(data: T) -> String { |
Contributor
There was a problem hiding this comment.
great, curious if that would be more performance if create a string directly from bytes without utf checks(which is redundant ) and extract casts?
let bytes = data.as_ref();
let mut out = Vec::with_capacity(bytes.len() * 2);
for &b in bytes {
out.push(HEX_CHARS_LOWER[(b >> 4) as usize]);
out.push(HEX_CHARS_LOWER[(b & 0x0f) as usize]);
}
unsafe { String::from_utf8_unchecked(out) }
Member
Author
There was a problem hiding this comment.
I tried this, but the difference is tiny, so it may not be worth it.
comphead
approved these changes
Dec 30, 2025
comphead
left a comment
Contributor
There was a problem hiding this comment.
Thanks @andygrove it makes sense to me
viirya
approved these changes
Dec 30, 2025
viirya
reviewed
Dec 30, 2025
Comment on lines
+163
to
+164
| /// Fast hex encoding using a lookup table instead of format strings. | ||
| /// This is significantly faster than using `write!("{:02x}")` for each byte. |
Member
There was a problem hiding this comment.
It's a bit weird to put the comment here. Maybe put on top of HEX_CHARS_LOWER lookup code.
Jefffrey
approved these changes
Dec 31, 2025
getChan
approved these changes
Dec 31, 2025
martin-g
approved these changes
Dec 31, 2025
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Jan 1, 2026
## Which issue does this PR close? - Part of #19569 ## Rationale for this change Completes the hex encoding optimization work from #19568 by replacing `write!` format strings with lookup tables in the remaining instances (`hex` and `sha1` functions in spark module). ## What changes are included in this PR? Avoid using `write!` with a format string and use a more efficient approach ## Are these changes tested? Yes ## Are there any user-facing changes? No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
"{b:02x}"to hex-encode bytes #19569Rationale for this change
I ran microbenchmarks comparing DataFusion with DuckDB for string functions (see apache/datafusion-benchmarks#26) and noticed that DF was very slow for
md5.This PR improves performance:
What changes are included in this PR?
Avoid using
write!with a format string and use a more efficient approachAre these changes tested?
Are there any user-facing changes?